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ABSTRACT 

Online Peer Instruction has become prevalent in many “llipped 
classroom’ settings, yet little work has been done to exam- 
ine the content students generate in such a learning envi- 
ronment. This study characterizes a datasot generated by 
‘an open-source, web-based homework system that prompts 
students to first answer questions, and then provide expla- 
nations of their reasoning. Of particular interest in this 
dataset, is that students are also prompted to evaluate a 
subset of peer explanations based on how convincing they 
fare, as part of the Peer Instruction learning seript. Since 
these student “votes” are then used in the selection of what is 
shown to future learners, we cast this as an instance of learn 
cersourcing, a paradigm that presents new research opportu- 
nities for the Learning Analytics community. This study 
characterizes a dataset from one Peer Instruction tool, that 
includes not only the student generated answers and expla- 
nations, but this novel “vote” attribute, which aims to cap- 
ture how convincing each explanation is to other learners. 
‘The dataset includes longitudinal observations of student ro- 
sponses over the course of a semester, following groups from. 
three STEM disciplines. The data is made available to in- 


terested researchers! 
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1. INTRODUCTION 

‘The effectiveness of Peer Instruction on learnin 
class, and the success of Intelligent Tutoring Systems and 
MOOCS outside of class, have in part, led to the devel 
‘opment of web-based platforms for asynchronous Peer In- 
struction [6][27]- Recently. other similar learning environ- 
ments have been developed. centred on having students ex- 
plain their reasoning, and then evaluate the explanations of 
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their peors[22|[5]. The increasing use of this form of online 
Tearning exercise implies that a new type of data is being 
generated, wherein lie opportunities to examine theories of 
hhow self explanation and comparative peer assessment may 
impact Iearning, 


Just-in-Time- Teaching environments, a web-based platform 
for asynchronous peer instruction can substantially reduce 
the time teachers” spend searching the data to identify stu- 
dent misconceptions. 

compare answers with peers, they receive a form of immed 
ate feedback on their own explanation. Last, when postin 
threads and sub-threads to large scale on-line discussions. 
such as MOOCS, an asynchronous Peer Instruction platform 
offers a more structured alternative and ties student expla- 
nations to an answer choice, allowing for more organized 
aggregation of ideas [2] 


Second, when students are asked to 


"These platforms open new research questions and opporti- 
nities for the Educational Data Mining community. First 
these systems eapture new modalities of data, specificaly, 
the written explanations for answer choices, which acts as 
proxy data: representing the cognitive reflections elicited in 
conversations students have with peers during small- group 
in-class Peer Instruction discussions. Second, these enviro 
‘ments introduce challenges common to any platform centred 
fon user generated content: quality control and recommenda- 
tion. The power of having students generating the explan: 
tions to different answer choices, and then rating them, e 
ables scaling up of technologies that facilitate flipped teae 
ing practices{14). However, once these tools do scale, sheer 
volume requires automatic approaches for filtering out low 
quality content. Once filtering is complete, recommendation 
place to most effectively help eur- 
rent students navigate the large volume of content generated 
by past students, with the ultimate objective of optimiz- 
ing individual learning gains. Further research is needed on 
learnersourced data sets so as to develop best practices that 
leverage the effectiveness of student written and ranked ex- 
planations for adaptive learning experiences, while avoidin 
the pitfalls that can lead to the valuable data drowning in 


2. OBJECTIVES 
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‘This paper characterizes a dataset generated inside one on- 
line platform for asynchronous Peer Instruction, with the 
‘im of identifying the potential research questions and lim- 
itations afforded by this novel application. The use of the 
tool is growing, now reaching over 50 course offerings across 
at least 5 undergraduate institutions in different science dis 
ciplines. The contexts are varied, but the common thread 
is that instructors are all using the tool as an attempt to 
increase in-class student engagement with pre-instructional 
quizzes, and tailoring their lectures based on the free-text 
explanations students provide for their answers. Thus, the 
ultimate goal of this study is two-fold: 


+ introduce a novel source of data to the Educational 
Data Mining research community. which has the po- 
tential to open new lines of inquiry based on th 
voting” attribute. Students not only write explana- 
tions to justify their answer choice to conceptual sei- 
fence questions, but they are asked to choose which of 
a subset of their peers’ explanations are most convine- 
ing, 

+ identify opportunities and challenges related to the de- 
sign of platforms that rely on learnersourced content 
such as choosing the most effective content to foster 
earning: filtering weak or irrelevant student explana- 
tions; categorizing and summarizing student explana 
tions for teacher reporting, in large classes. 


3. BACKGROUND AND RELATED WORK 
3.1. Peer Instruction 


‘The interactive engagement technique most relevant to our 
work here is Peer Instruction: a method for promoting class 
room discussion that has been shown to enhance learning [5) 
In this common classroom practice, teachers 


1. poll their students on a multiple-choice item, using 
some form of Audience Response System (e.g. elick- 
ers) 

2. collect the distribution of answers, and maybe even 
share back with the students, 

3. without revealing the correct answer, prompt students 
to explain their reasoning for their answer choice to 
fa peer nearby, ideally with someone with a diffe 
perspective 

4, re-poll the students after the small group discussions 


‘The platform at the centre of this study facilitates an asyn- 
chronous version of the above script 


3.2 Comparison-based peer assessment 
‘There are other systems similar in design to asynchronous 
peer instruction; they differ in that the items prompt for 
‘open-ended responses, as opposed to multiple-choice ques 
tions. However these systems still incide a similar review- 
step after submission of an answer, where students are asd 


to compare and evaluate the quality of the explanations sub- 
mitted by peers who had already answered the item, 


For example, in the ComPAIR system [22], students first 
submit their written answer toa prompt. ‘They are then 
shown pairs of their peers’ answers, prompted first to give 
feedback to each of the answers in the pair, and then choose 


fone as the better response. ‘The pairwise comparison at the 
heart of this tool leverages learners’ inherent ability to make 
judgments regarding an answer's quality relative to another: 
to make up for the lack of expertise usually needed to provide 
useful feedback on content in isolation. JuxtaPeer (5) is a 
similar system, where the pairwise comparisons are anchored 
fon one object at a time, and have been shown to improve the 
quality of feedback that peers can provide to one another. 


3.3 Explanations Datasets 
‘Two of the most prominent sources of learning analytics 
datasets ate from the ASSIStments platform[15], and PSLC 
DataShop(i9). They both provide significant contributions 
to Learning Analytics and Educational Data Mining revarchers, 
by making available a wide variety of data from different om 
Tine learning tools. They include datasets with free text re- 
sponses, including math hints generated by students in 
SiSTinents, and student explanations to science questions 
inside the Andes project (hosted in Datashop) 


casting student explanations as short arguments in favour 
of their answer choice, we can look to the Argumentation 
Mining research community for sample datasets. For ex- 
ample, a dataset of persuasive student essays that are fully 
annotated for argumentative relations was recently released 
[26]. The International Corpus of Learner English[12] is used 


extensively to model how students make arguments. 


Another sign of the growing int 
generated text are the Automated Essay [16] and 
Automatic Short Answer Scoring [17] competitions hosted 
fn the data science platform, kagsle.com. ‘These datasets 
are still freely available as well, 


im analyzing student 


‘To the best of our knowledge, none of the above data sources 
include all of the defining characteristics that are gen 
ated by online Peer Instruction, such as the student's initial 
answer choice and explanation, a student's second answer 
choice after having reviewed peer explanations, and most 
importantly, the peer explanation the student found most 
convincing, 


3.4 Learnersourcing explanations 
‘Web-based homework systems are effective because students 
get immediate feedback as to whether they answered cor- 
Fectly. However, as the number of question items grows, as 
wrens associated answer choices, generating high quality 
explanations that help different types of learners resolve dif 
ferent sets of misconceptions, s impractical for teachers (14) 
Moreover, explanations written by content experts may also 
suffer from the expert Bind spot, wherein their high level 
of familiarity with the subject matter actually might act 
ally make their explanations more dificult to understand to 
novices [20] 


‘The concept of learnersoureing is a sub-type of crowdsoure- 
ing, wherein domain novices contribute to the human com- 
putation workflow as part of their leaning process [28} 
Peet Wise [10] is an environment within which students make 
their own questions, and share them with peers, along with 
accompanying solutions. RiPPLE is a tool that follows the 
same model, but adds an adaptive recommendation engine 
[18].The AXIS system [20] prompts students to provide ex- 
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Figure 1: Asynchronous Peer Instruction - Step 
1, screenshot of student answering multiple-choice 
question, and explaining their thinking inside a text 
box 


planations to their answers, rate the explanations of their 
peers, and then machine learning to curate these to a con- 
stantly evolving set of explanations that optimize for pro- 
moting student learning. ASSISTments, another widely used 
learning platform, developed the Pecr ASSIST plugin (24), 
which asked students to write explanations to their answer 
submissions, to be sed as hints for future students, 


4. MYDALITE: PLATFORM AND DATA 
4.1 The Platform 

dalite-ng is an open-source project (25) that has been in ne- 
tive development sinew 2013, and has been used in MOOCS 
fas well as on campus course offerings. -myDALITE.org. is 
fone instance of this code-base, offered as a hosted ser 
that is free to all teachers and students. It is maintained by 
a network of learning science researchers and practitioners, 
‘whose mission is to promote the uptake of student-centred 
active learning pedagogical practices. Teachers sign up, asi 
thor their own questions, and distribute to their students 
at their discretion. The script for the student completing a 
‘question item in daliteeng is 


1. Question start: student is presented with a multiple. 
choice question. They are asked to choose an answ 
choice and enter a free text response to explain their 

reasoning. 
Question review: without indicating whether the 
Student chose the correct answer, the tool reflects biele 
explanation 
id to ro 


to the student their own ehoiee, aud the 
they just entered. ‘They are then prom 
consider their answer, by reacling the explanations of 
other students. Tn the top half of the page, they are 
shown up to 4 other explanations by students who 
chose the same answer choice. In the second half of 
the page, they are then shown up to 4 more expla- 
ations to a different answer choice. Students must 
indicate whic 


their second answer choice in this ro- 


Thin lens1 


Figure 2: Asynchronous Peer Instruction - Step 2 , 
screenshot of student choosing a peer's explanation 
of a different answerehoice 


view step, by selecting one of these explanations. They 
alo have the option of selecting their own explanation 
fs the most convineing. There are several fuctors that 
0 into the selection of what the students are shown 
heres 
‘if the student answered incorrectly on the first 
step, the explanations in the second balf of the 
page will be for the correct choice 
‘ifthe studont did in fact answer correctly on the 
first step, the explanations in the second half of 
the page will be for the most popular incorrect 
‘© There are two different heuristies for the selection 
of explanations for each answer eliiee: 
~ Random, which is useful for when a question 
is newly introduced to the datahase, and not 
enough students have answered to reliably e5- 
timate which answers are most convincing 
— preferentially selecting from explanations that 
have already been chosen as convincing 
3. Question summary The entire flow of information 
iS reflected buck to the student for review: their frst 
answer, their own explanation, their second answer 
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choice and the associated explanation that they chose 
as most convincing. The correct answer is also finally 
revealed. 


4.2. Data Collection 

‘The data in this study comes from the 2018-2019 academic 
year, wherein the platform was more heavily used than ever 
before, due to additional on-boarding support offered to 
teachers by the host network, All teachers who make ques 
tion items on the platform must release their content under 
Creative Commons licenses, and are made aware that the 
learning-data generated by the students in their groups may 
be used for academic research, Students are advised upon 
signing in, that their learning traces, in anonymized form, 
may be used for research, and that if they do not wish to 
share their data, they ean revoke their consent at any tinn 
without any impact on academic standing in their courses 
‘The data gathered for this study spans three STEM disei- 
plines where there happened to be the most activity: Bi- 
ology, Chemistry, and Physies. There are many different 
groups of students in Physics and Chemistry. each with a dif. 
ferent teacher (although all in undergraduate level courses) 
‘while all of the data in Biology comes from one large fresh 
man group that used the tool very heavily. In the ease of 
‘few groups, the items were assigned by teachers as op- 
tional, not-for-credit items, meant to provide extra practice 
study exercises (this information is provided in the meta 
data file of the dataset), For those eases when myDALITE, 
was used for eredit, students received 0.5 marks for choosing 
the correct answer on their first attempt, and 0.5 marks for 
choosing the correct answer choice after the review step. No 
credit was ever assigned based on a formal expert evaluation 
of the student explanations. 


4.3. Dataset 
Bach record in the dataset is comprised of the following 
fields: 


‘© anonymized student identifier 


‘© anonymized group/eourse identifier (with meta-data 
fon whether the activities were assigned for credit of 
not) 

‘© question prompt text (and any associated images) 

f Student's first answer choice 

‘© student's explanation for their first answer choice 

‘peer explanations shown to student on second step 

f student's second answer choice 

«© the poer explanation they selected as most convincing 


for their second answer choice 
‘© timestamps associated with 
— when the student first opened the problem 
— shen the student entered their first answer choice, 
‘and associated explanation 
— when the student entered their second answer choice, 
and associated peer explanation 


Certain filters were applied for the purposes of data extrac- 
tion for this study. The only groups that were retained were 
those having 10 students or mote, each of whom having an- 
sswered at least 10 questions. The only disciplines that were 
inchided in the curtent dataset were ones with over 10,000 


‘Table 1 


‘iplines 


Ny NN, 


Biology «1 ~—«SHG— 2821965387 
Physics 16 1250 57250286 40 
Chemistry 16 105558228819 27 


Table 2: Relative number 
from step 1 to step 2 


fred Be 


070 0.10 001 0.09 040 
0.79 0.09 0.0L 0.07 O44 
069 0.12 0.01 0.11 0.38 


Bioloay 
Physics 
Chemistry 


5. DESCRIPTIVE STATISTICS 
‘As can be seen in Table 1, there are relatively similar mum- 
bers of responses across the three disciplines 


umber of groups 
umber of unique students 

umber of items 

total number of answers by all students 

average number of items completed by each student 


‘This table demonstrates the valuable longitudinal nature of 
the dataset, in that across the disciplines, there are, on av- 
rage, more than 25 observations per student, which could 
help building a more robust learner models. 


In Tuble 2, we see the proportion of times students changed 
their answers on the answer review step. 


# LC : fraction of responses where students chose the 
correct answer choice on their first attempt 

# A: fraction of responses where students switch their 
answer choice on review step 

— Yo: total fraction of answers where students changed 
their answer choice from step 1 to step 2 

=r > w: fraction of responses where students 
switeh their answer choice on review step, going 
from right to wrong. 

—w > r= fraction of responses where students 
switch their answer choice on review step, g0- 
ing from wrong to right, presumably after reading 
their peers’ explanations 

# Ac: fraction of responses when students do not change 
their answer choice on review step, but choose an ex- 
planation other than their own as most convincing 


Across the disciplines, the items in this dataset are easy 
enough for students to choose the correct answer choice on 
their first attempt almost three out of four times. The expla- 
nations of their peers are almost never able these convince 
students to switeh from the right answer choice to a wrong. 
fone. However of the students who choose the wrong answer 
fon their first attempt, after having access to the explanation 
of their peers, these students switch to the correct answer 
choice at the review step almost one out of three times. These 
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Figure : Visualizing student transitions in asyn- 
chronous peer instruction with a Sankey diagram 


‘Table 8: Descriptive statistics on explanation word 
counts 


WE WO<5 Acuna 


Biology 7 og 0.19 
Physies 1 0A2 08 
Chemistry 19 0.1908 


relative transitions are visualized i the Sankey diagearn ia. 
Figure 3, 


In table 3, 


« TT is the average word count of the explanations 

© IC < Sis the proportion of explanations that have 
Jess than 5 words 

# Aciangest the proportion of times students selected the 
peer explanation that had the most words amongst 
those that they wore shoven 


Hore in Table 3, we see that many students write explana- 
tions that are too short to form a sentence in the Biology 
subset, and that even in the other disciplines. the explana- 
tions are not long-form persuasive essays, but likely claser to 
short answers, However, stucents seem to show a preference 
for explanations that are longer in length when “voting” for 
the most convincing explanation on the review step. 


6. DISCUSSION 

Learnersourcing shows immense potential for sealing 1p on- 
line Peer Instruction, but also presents new challenges com- 
‘mon to contexts centred on user-generated content. 


A quick sampling of the large number of explanations with 
less than § words likely indicates that students do not see 
the value of writing explanations, unless they will reesive 
‘course credit for the task, Work from the argument mining, 
community may be useful here to automatically assess the 
quality of explanations. Under study is the impact of wob- 
‘based reputation systems on increasing student engagement, 
which have been shown to increase engagement in learn 
ing environments by offering virtual achievement. rewards, 


such as badges and Ieaderboards|9]. Another open research 
question is in automatic quality control, given that the first 
fow students who complete a question, and submit an ex 
planation, will have their work shown to many subsequent 
students." Work that has been done on automatic filtering 
[11] of explanations based! on unsupervised clustering could 
prove henelicial here, 


‘The value and uniqueness of this dataset remains in the 
“voting” data: modelling what Einguistie properties and con- 
ceptual constructs students find convincing, in the Ianguage 
of their peor, is fertile ground for research. The longitudinal 
data also allows for modelling the evolution of how students 
start intograting domain specific concepts into their expla- 
nations across a semester, as well as “voting” for them in the 
peer-explanations they find most convincing. 


6.1 Future Work 

‘Work must now be done on better understanding how to op- 
timize the heuristics that solect what peer explanations are 
shown to students in order to enbiance learning, This will = 
quire building studont models of ability and models of item 
difficulty. The linguistic properties are also of koy interest: 
can this mode of comparative peer assessment data be used 
to inform our models of whether students have attained do- 
rain literacy? Finally, how do such environments promote 
student engagement in flipped classroom contexts? We look 
forward to collaborating with the community through this 
novel source of data to along these lines of research, 


‘Many of the design implementation decisions for these plat- 
forms are made with pragmatic motivations in mind and 
nove to be bettor informed by learning analytics theory. The 
platform at the center of this study is a model to examine 
more closely also because it is an open-source project, de- 
veloped as part of Research Practice Partnership [7], where 
learning analytics researchers are atively working, with ine 
structors using the tool to better align teaching practices 
with sound pedagogical design 
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